Collaborative and quality control scoring system and method

ABSTRACT

A method and system for managing resolvers of test answers to maintain consistent and efficient scoring of test answers. Collaborative scoring allows two test resolvers to score the same test answer and resolves potential discrepancies in the scores provided by the resolvers. Quality items monitor a test resolver&#39;s scoring performance against a known standard. A quality-control test item, having a known score, is transmitted to a test resolver. After receiving a score from the resolver, the known score is compared with the resolver&#39;s score and a quality profile is generated for the resolver.

This is a continuation of application Ser. No., 09/003,979, filed onJan. 7, 1998, which is a continuation of application Ser. No.08/561,081, filed Nov. 20, 1995, which is now U.S. Pat. No. 5,735,694,which is a continuation of application Ser. No., 08/290,014, which isnow U.S. Pat. No. 5,558,521 filed Aug. 12, 1994, which is a division ofapplication Ser. No., 08/014,176, now U.S. Pat. 5,437,554, filed Feb. 5,1993. U.S. Pat. No. 5,735,694, U.S. Pat. No. 5,735,694, and U.S. Pat.5,437,554 are hereby incorporated by reference in their entirety.

FIELD OF THE INVENTION

The present invention relates to a system for processing answers to testquestions.

BACKGROUND OF THE INVENTION

The scoring of test answer sheets involves complex problems. These testanswer sheets typically include a series of response positions such as,for example, “bubbles,” ovals, or rectangles. A person taking a testwould, for example, darken in an appropriate oval with a pencil toanswer a multiple choice question. These test answer sheets may alsoinclude handwritten answers, such as essay or short answer questions.Systems for scanning and scoring the bubbles on such answer sheets areknown in the art. Increased difficulties are encountered, however, whensuch answer sheets either include other types of answers, such ashandwritten answers, or cannot be machine graded. For example, if thestudent has failed to include his or her name on the test answer sheet,the system may be unable to machine score the test answer.

The goals in scoring test answers that cannot be machine scored includeefficiency and consistency. These test answer sheets are typicallyscored by test resolvers either by manually scoring the physical testanswer sheet or scoring an electronic representation of the test answersheet on a computer. Ideally, the scores provided by the various testresolvers for a particular test question should be consistent, since thescores are used in comparing performance of the students against oneanother. In addition, a test resolver should ideally work efficiently soas to maintain consistently high scoring rates. The test resolver shouldnot have such a high scoring rate that the consistency or quality ofscoring significantly declines; likewise, the test resolver should nothave such a low scoring rate that the too few answer sheets are beingscored. This manual scoring of test answer sheets, however, makes itdifficult to monitor the consistency of scoring among the various testresolvers.

In many situations, test resolvers actually travel to a particularlocation so that all test resolvers may simultaneously score test answersheets. Requiring the test resolvers to travel to a given location isinconvenient for the resolvers and expensive for those who administerthe tests. Furthermore, tracking the performance of test resolversagainst both their own performance and the performance of otherresolvers can be very difficult with a manual scoring environment.

The process of resolving test questions is currently done manually, andthis presents problems. A resolver is manually presented with the actualtest answer sheets for scoring. This process is relatively inefficient,since the resolvers must score the answer sheets one at a time and inthe order in which they are presented. Also, manual scoring systems donot have the capability to efficiently gather and categorize the testanswers for subsequent analysis. Therefore, with a manual system it isvery difficult to determine how teaching methods should be changed todecrease, for example, the number of incorrect answers.

A need thus exists for a system that promotes and achieves consistencyand efficiency in scoring or resolving of tests.

SUMMARY OF THE INVENTION

The present test resolver management system facilitates consistent,accurate, and high quality scoring of test answers. A collaborativescoring system and method allows two test resolvers to score the sametest answer and resolves potential discrepancies in the scores providedby the resolvers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a network that incorporates the presentinvention.

FIG. 2 is a block diagram of a portion of the network shown in FIG. 1.

FIG. 3 is a block diagram of the scanning configuration in the networkof FIG. 1.

FIG. 4 is a block diagram of the server in the network of FIG. 1.

FIG. 5 is a flow chart of receiving and processing of test items.

FIG. 6 is a flow chart of multiple item scoring.

FIG. 7 is a flow chart of categorized (special) item reporting.

FIGS. 8-10 are a flow chart of collaborative scoring.

FIG. 11 is a flow chart of quality item processing.

FIG. 12 is a flow chart of resolver monitoring and feedback.

FIG. 13 is a flow chart of an on-line scoring guide system.

FIG. 14 is an example of a user interface for use with multiple itemscoring.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

In the following detailed description of the preferred embodiment,reference is made to the accompanying drawings which form a part hereofand in which is shown by way of illustration a specific embodiment inwhich the invention may be practiced. This embodiment is described insufficient detail to enable those skilled in the art to practice theinvention, and it is to be understood that other embodiments may beutilized and that structural or logical changes may be made withoutdeparting from the scope of the present invention. The followingdetailed description is, therefore, not to be taken in a limiting sense,and the scope of the present invention is defined by the appendedclaims.

HARDWARE CONFIGURATION

FIG. 1 illustrates an example of a hardware configuration for a networkthat incorporates the present invention. This configuration is shown asan example only; many different hardware configurations are available,as recognized by one skilled in the art, for implementing the softwareprocessing functions described below. The network shown comprises amainframe computer 20 interfaced through a backbone token ring to aplurality of RISC servers 11, 12 and 13. Each RISC server is interfacedto a token ring that contains work stations and scanners. The RISCserver 11 is connected in token ring 1 to scanners 14 and work stations19. The RISC server 12 is connected in token ring 2 to scanner 15 andwork stations 18. The RISC server 13 is connected in token ring 3 toscanners 16 and work stations 17. The mainframe computer 20 is alsoconnected to a high capacity printer 22 and a low capacity printer 21for printing reports of stored data within the system.

The system uses the scanners for reading in test answer sheets. Thesetest answer sheets may comprise, for example, test forms with “bubbles”or ovals representing possible answers, handwritten essays, or othervarious types of written or printed information. After receiving thescanned test data, the system within the RISC servers can process thosescanned test answer sheets to generate test items of interest from theanswer sheets. A test item is, therefore, an electronic representationof at least a portion of a test answer sheet. The system may distributethese test items to the work stations for on-line scoring. A test scorerat a work station can then score the test item and enter a test score.The system receives the test scores via the network and the RISC serversand distributes the scores to an appropriate computer for subsequentprinting and reporting; the appropriate computer may include, forexample, the mainframe computer 20 or a server. The system may alsotransmit the test scores to, for example, a disk or telephone line.

FIG. 2 is a more detailed block diagram of a portion of the networkshown in FIG. 1. As shown in FIG. 2, the scanning units shown in FIG. 1typically comprise a scanner 25 interfaced to a computer 24 and personalcomputer (PC) 26. FIG. 3 shows a more detailed block diagram of ascanning unit. The scanner 25 contains a camera 31 for optically readingin a test answer sheet, and further contains optical mark recognition(OMR) logic 32 for processing the scanned data received from camera 31.The PC 26, preferably implemented with a high performance 486-level PC,contains a frame buffer 23 for receiving the scanned image data from thescanner 25.

The computer 24, preferably implemented with an HP 1000, is interfacedto the scanner 25 and PC 26 for controlling the operation of thescanning unit. The computer 24 is optional; the system may alternativelybe configured such that all of the functionality of the computer 24 iswithin the PC 26. The computer 24 controls the scanner via the OMR logic32 and thus controls when image data is scanned in and subsequentlytransferred to the PC 26. The PC 26 essentially acts as a buffer forholding the image data. The computer 24 further controls when the PC 26will interrogate the image data for transmission to a server 27 forsubsequent processing and scoring. The PC 26 can also electronicallyremove or “clip” an area of interest from the image data, whichrepresents at least a portion of the scanned test answer sheets.

Examples of two systems for storing and extracting information fromscanned images of test answer sheets are shown in U.S. Pat. Nos.5,134,669 and 5,103,490, both of which are assigned to National ComputerSystems, Inc. and are incorporated herein by reference as if fully setforth.

The server 27 receives the image data, which includes test items, andprovides for processing and control of the image data. This portion,which may be a test item, is then distributed to the work stations 28,29 and 30 for subsequent scoring. A test resolver (scorer) at the workstation typically receives the test item, performs the scoring, andtransmits the score to the receiving computer.

FIG. 4 is a block diagram of the hardware and software functions in aserver in the network of FIG. 1. A scan control module 31 interfaceswith the scanner PC 26 and receives the image data. The image data isstored in a raw item database 36. The central application repository(CAR) 33 typically stores document definitions and handling criteria.The document process queue 37 functions as a buffer into a mainprocessing module 45 in server 27.

The main processing module 45 controls the processing of test items. Itcontrols the transmission of test items to the work stations for scoringand the transmission of scores to the mainframe computer 20. The mainprocessing module 45 also monitors the performance of the test resolversto maintain consistent and efficient resolving of test items, as isexplained below.

The main processing module 45 typically contains the following basicfunctions, which are controlled by system management module 32. A workflow module 38 receives image data from the database 36 and controls theflow of data into an edit process module 39. The edit process module 39may perform machine scoring of the test items. For those test itemswhich cannot be machine scored, or possibly for other test items, thesystem transmits such test items to the job build function 40. The jobbuild function 40 determines what type of subsequent scoring is requiredfor the test item and, for example, which work station will receive thetest item. A job send module 41 receives the test item and transmits itto a router 42, which in turn transmits the test item to a send/receivecommunication module 43. Edit work module 34 and edit server module 35control the flow of test items into and out of server 27. Incoming data,such as test answers from the work station, are transmitted throughmodules 34 and 35 to a job receive module 44. The job receive moduletransmits the data to the edit process module 39 for subsequent storagewithin the database 36.

SOFTWARE PROCESSING

FIG. 5 is a flow chart of typical scanning and processing of test andanswer sheets. The document processing system receives the test answersheets, or other documents, at step 50 and performs initial clericalpreparation of the documents (step 51) for scanning at step 52. Thesystem at step 52 scans the documents for OMR and other image data. Thesystem may then process the OMR bubbles at step 53 and store the data inthe work-in-process storage (WIP) at step 54. The system at step 56 can“clip” areas of interest from the scanned image. The step of “clipping”involves electronically removing, typically in software, a portion ofthe test item or scanned image. These “clipped” areas may comprise anyportion of a test answer sheet; for example, a handwritten essay orselected response positions. The system may also receive image datadirectly from foreign sources, magnetic or electronic, and store thedata in raw item database 36. Subsequent operations on the data are thesame regardless as to the source of the data. After “clipping” areas ofinterest from the image, the system stores the test items at step 57 inthe work-in-process storage 55.

The system waits at step 55 until it determines that a test item isready to be resolved or scored. If multiple resolution items are presentwithin the image data, as determined at step 59, then the system sendsthe test item to multiple item processing at step 63. Otherwise, thesystem performs other resolution processes on the data at step 60 andstores the result in work-in-process storage 55 at step 61. Otherresolution processes may include, for example, machine scoring, raw keyentry, and analytic resolving.

Analytic resolving or scoring may include, for example, map comparisonssuch as bit-mapped comparisons between two test items. The mapcomparisons allow a test resolver to compare, for example, the answersof a respondent over time to track the respondent's progress. Forexample, the analytic scoring may involve comparing two hand-drawncircles by the respondent to determine if the respondent's accuracy indrawing circles has improved over time. Analytic scoring may alsoinclude, for example, circling or electronically indicating misspelledwords and punctuation errors in an answer such as an essay.

Multiple Item Scoring

FIG. 6 is a flow chart of typical multiple item processing. The systemat step 64 typically first fetches a multiple item image from thework-in-process storage. The image is stored in a multiple item displaymemory 65 and a multiple item display storage 69 for subsequent displayto a resolver. The system continues to receive multiple items untileither the item display is full, as determined at step 66, or no moremultiple items are present as determined at step 68. As long as thedisplay is not full and additional multiple items are present, thesystem preferably scans the work-in-process storage at step 67 foradditional items. When the multiple item display is full or no moremultiple items are present, the system sends the compiled multiple itemsto a resolver at step 70 and displays the multiple test items on theresolver display 71.

The system typically transmits test items to a particular resolver basedupon the resolver's resolution expertise. For example, a certainresolver may be assigned to score all of the test items relating toscience questions. Resolution expertise may also comprise, for example,math, english, history, geography, foreign languages, or other subjects.

An example of an interface on the resolver display is shown in FIG. 14.The interface typically comprises a plurality of cells 74, with eachcell containing one test item to be resolved. After displaying themultiple items in the cells of the resolver display, the system allowsthe resolver at step 72 to score the multiple items. A test resolverwould typically indicate the score of the answers by using a “mouse,”light pen, touch screen, voice input, or some other type of cursorcontrol or input device.

In the example shown in FIG. 14, the correct answer is “four” and theincorrect answers are indicated by the shading. Alternatively, aresolver could indicate the correct answers. The advantage of themultiple item system arises from the simultaneous display of test itemsin the cells 74, which allows a test resolver to quickly score many testitems and thus achieve a faster response time in comparison to thedisplay and scoring of only a single test item at a time. Even thesimultaneous display of two items increases response time. As the matrixof cells increases, the simultaneous display of test items achieves asignificant increase in response time and resolver attention and focus.

After scoring or resolving, the system receives the results at step 73for subsequent storage in work-in-process storage 55. A test resolvertypically transmits the results of resolving all displayed test items inthe cells as a single unit for batch processing.

Categorized Item Reporting

FIG. 7 is a typical flow chart of categorized (special) item reporting.Categorized item reporting allows the system to both group answersaccording to predefined categories and monitor processes used by thestudents or test-takers in arriving at a given answer. The categories inwhich test answers may be grouped include, for example, incorrectanswers and correct answers within a curriculum unit within aninstructional grouping and requested time frames; for example, all ofthe incorrect math answers in a particular instructor's class during theprevious school year. Other groupings are possible depending upon theneeds of the test resolvers and instructors who teach the material towhich the test relates.

In addition, the system may merge an image of a test item with thecorresponding score. In order to facilitate teaching of material towhich the test relates, the system typically merges a test itemrepresenting an incorrect answer with the corresponding score. Byreporting the actual test item, an instructor may gain insight into athought process used by the student in arriving at the incorrect answer.Therefore, by having some knowledge of why a student answered a testquestion incorrectly, an instructor can take measures to change ormodify teaching strategies to correct the situation.

The categorized item reporting normally comprises the followingfunctions. The system at step 75 scans the work-in-process storage foritems that are ready to be reported. If test items are ready forreporting, as determined at step 76, the system processes the data atstep 77 for generating an appropriate report of the data. At step 78,the system scans the central application. repository for definitions ofcategorized (special) items. As special items are available forreporting, as determined at step 79, the system retrieves the specialitems at step 80 and can merge it at step 81 with other reportinformation, such as the corresponding test items, as explained above.The system then distributes a report at step 82, which can be a printedreport.

Collaborative Scoring

FIGS. 8-11 are a flow chart of a typical collaborative scoring system.The collaborative scoring system provides for functions to achievefairness and objectivity in resolving of test items. The collaborativescoring, for example, allows two resolvers to score the same item and,if the answers are not within a certain predefined range, provides forsubsequent processing to resolve the discrepancy.

The system at steps 83 and 84 determines if items are available forscoring. At step 85, the system receives collaborative scoringrequirements from the database and determines at step 86 ifcollaborative scoring is required. Examples of collaborative scoringrequirements are illustrated below. If collaborative scoring has beenspecified, the system retrieves the item to be scored from thework-in-process database at step 87 and sends the item to resolvers 1and 2 at steps 88 and 91. At step 140, the system can prevent resolver 1and resolver 2 from scoring the same test item if they have averagescores within predefined scoring criteria.

The system is further able to choose resolvers according to selectioncriteria at steps 89 and 90. The selection criteria of the resolvers forscoring answers may include, for example, race, gender, or geographiclocation. The ability of the system to assign test resolvers to scoreparticular test items provides the basis for increased fairness andconsistency in the scoring of tests. For example, the system may assigntest resolvers to test items based on the same racial classification,meaning that the test resolver has the same racial classification as thestudent or respondent whose test the resolver is scoring. The system mayalso assign test resolvers to test items based on a different, forceddifferent, or preferred blend of classifications. The system monitorsconsistency in scoring based on the selection criteria and, moreimportantly, can change the selection criteria to ensure consistent andfair scoring of test items.

FIGS. 9A and 9B are a flow chart showing additional typical functions ofthe collaborative scoring. At steps 92 and 93, the system displays theitems to resolvers 1 and 2 for scoring. The system may further track theaverage scores of resolvers and not send the same test item to tworesolvers who have provided average scores within a predefined range.This also helps to achieve consistency in scoring. For example, if twoscorers each have provided high average scores in the past, asdetermined by the system, these two scorers should preferably not becollaboratively scoring the same test items, since it could result in“inflated” scores for particular test items.

The system records the scores from resolvers 1 and 2 at steps 94 and 95,respectively, and stores such scores in a temporary storage 96. At step97, the system compares the scores according to criteria specified inthe central application repository. Such criteria may include, forexample, requiring that the scores be within a predefined percentage ofeach other. If the scores meet the criteria as determined at step 98,the system records the score in the work-in-process database at step 46.At step 141, the system determines if collaborative scoring is required.If not, the system records a score based on predefined criteria (step142), which may include a greater of the first and second scores (143),a lower of the first and second scores (144), and an average of thefirst and second scores (145). Otherwise, if the scores do not meet thecriteria, the system determines at step 99 if the scores of theresolvers must agree. If the first two resolvers scores do not need toagree, then the system preferably transmits the test item to a thirdresolver to “cure” the discrepancy in the first two scores. At step 100,the system determines if the third resolver should see the first scores.

FIG. 10 shows additional typical processing of the collaborativescoring. If the original resolvers 1 and 2 must agree on a score, thenthe system executes steps 101-105. The system then typically firstdisplays to each resolver the other resolver's score at steps 101 and102 so that each resolver can see the score determined by the otherresolver. At step 103, the system establishes a communication betweenthe two resolvers. Such a communication link may be, for example, anelectronic mail link so that the scorers can exchange informationregarding the reasoning behind the score provided. At step 146, thesystem signals resolver 1 and resolver 2 to interactively determine andenter a joint score. At step 104, the resolvers work together todetermine a single agreed-upon score for the test item. The system mayprevent the resolvers 1 and 2 from receiving another test item untilthey have entered an agreed-upon score for the previous test item (step147). Finally, at step 105, the system stores the agreed-upon score inthe work-in-process database.

Instead of allowing the resolvers to work together to record anagreed-upon score, the system may optionally record, as describe above,either a greater value of the first and second test scores, a lowervalue of the first and second test scores, or an average value of thefirst and second test scores.

If the collaborative scoring criteria specifies that the third resolvershould arbitrate the discrepancy and determine a score, then the systemdisplays scores from the resolvers 1 and 2 at step 106 for resolver 3.The third resolver (resolver 3) then typically enters a score for thetest item at step 107, and the system records the score in thework-in-process database at step 108.

If the collaborative scoring requirement specifies that the thirdresolver should not see the first two scores, then the system executessteps 109-111 and 148. At step 109, the system displays the test itemfor the third resolver. At step 148, the system prevents the third testresolver from viewing the first and second test scores. The thirdresolver then typically enters a score at step 110, and the systemrecords the score in the work-in-process database at step 111.

Quality Items

FIG. 11 is a typical flow chart of the use of quality items in thescoring process. The system can use quality items to check and monitorthe accuracy of the scoring for selected test resolvers in order tomaintain consistent and high quality scoring of test items. At step 112,the system determines or receives the quality criteria. The qualitycriteria may be, for example, a predetermined test item with a known“correct” score.

The system then waits for a scheduled quality check at step 113. At thequality check, the system, at step 114, sends the known quality item tothe scheduled resolver. At step 116, the system updates the resolver'squality profile based on the evaluation at step 115. If the resolvershould receive a quality result, as determined at step 117, the systemdisplays the quality profile to the resolver at step 118. At step 119,the system sends the quality profile to a manager for subsequent review.At step 120, the system takes action required to assure scoringaccuracy.

Resolver Monitoring and Feedback

FIG. 12 is a flow chart of typical resolver monitoring and feedback. Theprimary factors in monitoring performance typically include: (1)validity; (2) reliability; and (3) speed. In monitoring these factors,the system promotes repeatability of scoring. These factors may bemonitored by tracking a resolver's performance against past performanceof the resolver or against some known goal.

Validity is typically measured by determining if a particular resolveris applying the scoring key correctly to test items or, in other words,scoring test items as an expert would score the same items. Reliabilityis typically measured by determining if a particular will resolve thesame test item the same way over time (providing consistent scoring).Speed is typically measured by comparing a resolver's scoring rate withpast scoring rates of the resolver or other scoring rates, such asaverage scoring rates or benchmark scoring rates.

At step 121, the system typically continually monitors the resolver'sperformance and updates the performance. Monitoring the resolver'sperformance may include, as explained above, monitoring the resolver'svalidity, reliability, and speed in resolving test items. The systemperiodically, according to predefined criteria, performs performancechecks of the test resolvers. Predefined criteria may include, forexample: a time period; recalls (how often a resolver evaluates his orher own work); requesting help; the number of agreements among multipleresolvers; the amount of deviation between the resolver's score and aknown score, which may be determined using quality items; the frequencyof these deviations; the speed at which a resolver enters a responseduring resolving of test items; the length of time between scoresentered by a test resolver; a test resolver's previous scoring rate, anaverage scoring rate of a test resolver; average scoring rates of othertest resolvers; or some predetermined benchmark scoring rate.

At step 122, the system determines whether it is time for a scheduledperformance check according to the predetermined criteria. If it is timefor a performance check, the system at step 123 compares the resolvers'current performance, as determined at step 121, with the storedperformance criteria. At step 124, the system determines if there is adiscrepancy in the resolver's performance according to the predeterminedcriteria. For example, the system may determine if the resolver'scurrent scoring rate is within a predefined percentage of the averagescoring rate in order to ensure efficient scoring by the test resolver.If there is no discrepancy, the system returns to monitoring theresolver's performance. In addition, the system may store the resolver'scurrent performance values for later processing. Otherwise, the systemreports the discrepancy at step 125.

At step 126, the system determines if it should recommend a break inscoring to the resolver. If according to predetermined performancecriteria, the system should recommend a break in scoring, then thesystem signals the resolver at step 128 to halt scoring. Predefinedperformance criteria may include, for example, deviations in theresolver's validity, reliability, or speed of resolving test items.Examples of predefined performance criteria are provided above withrespect to the monitoring of resolvers' performance.

When the resolver stops scoring, the system may provide the resolverwith the option of requesting diversionary activities. Diversionaryactivities are designed to provide the test resolver with a rest periodand “break” from scoring to increase efficiency. Examples ofdiversionary activities include computer games and cross word puzzles.If the resolver has requested such diversionary activities, asdetermined at step 129, then the system transmits a diversionaryactivity to the resolver at step 130. Otherwise, the system returns tomonitoring the resolver's scoring rate when the resolver resumes thescoring.

If the system at step 126 does not recommend a break in scoring based onthe discrepancy, then the system may optionally provide the resolverwith diversionary activities as determined at step 127. If the resolvershould receive the diversionary activities, then the system sends suchactivities to the resolver at step 130. Otherwise the system returns tomonitoring the resolver's scoring rate.

On-Line Scoring Guide

FIG. 13 is a flow chart of a typical on-line scoring guide system. Theon-line scoring guide increases scoring efficiency by allowing theresolver to request scoring rules in order to assist in scoring aparticular test item. In response to the request, the system displaysscoring rules corresponding to a test question for the test itemcurrently displayed to the resolver. A resolver may thus quickly havespecific scoring rules available on-line while scoring test items. Thispromotes scoring efficiency and reduces unnecessary break timesresulting from determining how to score a particular test item.

At step 131, the system sends a test item to a resolver for scoring anddisplays the test item at step 132. If the resolver has requestedscoring rules, as determined at step 133, then the system interrogates astored scoring guide to locate scoring rules that correspond to a testquestion for the test item currently displayed to the resolver. Thesystem retrieves those particular scoring rules at step 135 and displaysthem to the resolver at step 136. The system preferably uses amulti-tasking environment in order to simultaneously display the scoringrules and the test item. At step 134, the system waits for the resolverto score the test item. At step 137, the system stores the test scoreentered by the resolver into the work-in-process storage.

As described above, the present invention is a system that processestest items. The various functions used in processing the test itemspromote efficient, high quality, and consistent scoring of test items.

While the present invention has been described in connection with thepreferred embodiment thereof, it will be understood that manymodifications will be readily apparent to those skilled in the art, andthis application is intended to cover any adaptations or variationsthereof. For example, a different hardware configuration may be usedwithout departing from the scope of the invention and many variations ofthe processes described may be used. It is manifestly intended that thisinvention be limited only by the claims and equivalents thereof.

What is claimed is:
 1. A system for verifying test answers between twoor more test resolvers, the system comprising: a) first receive meansfor electronically receiving an answer to a test question; b) transmitmeans for electronically transmitting the answer to first and secondtest resolver terminals to be scored by first and second test resolvers;c) second receive means for electronically receiving first and secondtest scores for the answer, as determined by the first and second testresolvers, from the first and second test resolver terminals; and d)verification means for electronically comparing the first and secondtest scores and for selectively recording a value related to the firstand second test scores if the first and second test scores did notcontain a discrepancy based on predefined criteria.
 2. The system ofclaim 1 further comprising display means for electronically displayingthe test question to the first and second test resolvers at the firstand second test resolver terminals.
 3. The system of claim 1 wherein theverification means farther comprises collaboration means for providingan electronic communication link between the first and second testresolver terminals and, if the first and second test scores contain thediscrepancy based on the predefined criteria, for signaling the firstand second test resolvers to interactively determine and enter a jointscore using the electronic communication link, the verification meansfurther comprising recording means for recording the joint score.
 4. Thesystem of claim 3 wherein the collaboration means further comprisesmeans for tracking average scores of the first and second test resolversand for preventing the first and second test resolvers from scoring asame test item if the first and second test resolvers each have averagescores within predefined scoring criteria.
 5. The system of claim 3wherein the verification means further comprises means for preventingthe first and second test resolver terminals from receiving anotheranswer to be scored until the first and second test resolvers haveentered the joint score.
 6. The system of claim 3 wherein theverification means further comprises means for transmitting the testquestions to a third test resolver terminal to be scored by a third testresolver, if the first and second test resolvers do not enter the jointscore within a predetermined time period.
 7. The system of claim 1wherein the verification means further comprises means for transmittingthe answer to a third test resolver terminal to be scored by a thirdtest resolver, if the first and second test scores contain thediscrepancy based on the predefined criteria, and for recording a thirdtest score, as determined by the third test resolver, received from thethird test resolver terminal.
 8. The system of claim 7 wherein the meansfor transmitting the answer to a third test resolver terminal furthercomprises means for providing the third test resolver terminal with thefirst and second test scores.
 9. The system of claim 7 wherein the meansfor transmitting the answer to a third test resolver terminal hercomprises means for preventing the third test resolver from viewing thefirst and second test scores on the third test resolver terminal beforethe third test resolver records the third test score.
 10. The system ofclaim 1, wherein the verification means further comprises means forelectronically linking the first and second test resolver terminals whenthe first and second test resolvers are located in differentgeographical locations.
 11. The system of claim 1 wherein theverification means comprises means for selectively recording, as thevalue, a greater of the first and second test scores, a lower of thefirst and second test scores, and an average of the first and secondtest scores.
 12. A method for verifying test answers between two or moretest resolvers, the method comprising the steps of: a) electronicallyreceiving an answer to a test question; b) electronically transmittingthe answer to first and second test resolver terminals to be scored byfirst and second test resolvers; c) electronically receiving first andsecond test scores for the answer, as determined by the first and secondtest resolvers, from the first and second test resolver terminals; andd) electronically comparing the first and second test scores and forselectively recording a value related to the first and second testscores if the first and second test scores did not contain a discrepancybased on predefined criteria.
 13. The method of claim 12 furthercomprising the step of electronically displaying the test question tothe first and second test resolvers at the first and second testresolver terminals.
 14. The method of claim 12 wherein the step ofelectronically comparing the first and second test scores furthercomprises the steps of: a) providing an electronic communication linkbetween the first and second test resolver terminals and, if the firstand second test scores contain the discrepancy based on the predefinedcriteria, signaling the first and second test resolvers to interactivelydetermine and enter a joint score using the electronic communicationlink; and b) recording the joint score.
 15. The method of claim 14further comprising the step of tracking average scores of the first andsecond test resolvers and preventing the first and second test resolversfrom scoring a same test item if the first and second test resolverseach have average scores within predefined scoring criteria.
 16. Themethod of claim 14 wherein the step of electronically comparing thefirst and second test scores further comprises the step of preventingthe first and second test resolver terminals from receiving anotheranswer to be scored until the first and second test resolvers haveentered the joint score.
 17. The method of claim 14 wherein the step ofelectronically comparing the first and second test scores furthercomprises the step of transmitting the test item to a third testresolver terminal to be scored by a third test resolver, if the firstand second test resolvers do not enter the joint score within apredetermined time period.
 18. The method of claim 12 wherein the stepof electronically comparing the first and second test scores furthercomprises the steps of: a) transmitting the answer to a third testresolver terminal to be scored by a third test resolver, if the firstand second test scores contain the discrepancy based on the predefinedcriteria; and b) recording a third test score, as determined by thethird test resolver, received from the third test resolver terminal. 19.The method of claim 12 wherein the step of transmitting the answer to athird test resolver terminal further comprises the step of providing thethird test resolver terminal with the first and second test scores. 20.The method of claim 12 wherein the step of transmitting the answer to athird test resolver terminal further comprises the step of preventingthe third test resolver from viewing the first and second test scores onthe third test resolver terminal before the third test resolver recordsthe third test score.
 21. The method of claim 12, wherein the step ofelectronically comparing the first and second test scores furthercomprises the step of electronically linking the first and second testresolver terminals when the first and second test resolvers are locatedin different geographical locations.
 22. The method of claim 12 whereinthe step of electronically comparing the first and second test scorescomprises the step of selectively recording, as the value, a greater ofthe first and second test scores, a lower of the first and second testscores, and an average of the first and second test scores.
 23. A systemfor verifying test answers between two or more test resolvers, thesystem comprising: a) a receiving subsystem configured to electronicallyreceive an answer to a test question; b) a transmit subsystem configuredto electronically transmit the answer to first and second test resolverterminals to be scored by first and second test resolvers; c) a testscore subsystem configured to electronically receive first and secondtest scores for the answer, as determined by the first and second testresolvers, from the first and second test resolver terminals; and d) acomparison subsystem configured to electronically compare the first andsecond test scores and to selectively recording a value related to thefirst and second test scores if the first and second test scores did notcontain a discrepancy based on predefined criteria.
 24. The system ofclaim 23 wherein the receiving subsystem is further configured toelectronically display the test question to the first and second testresolvers at the first and second test resolver terminals.
 25. Thesystem of claim 23 wherein the comparison subsystem is furtherconfigured to: a) provide an electronic communication link between thefirst and second test resolver terminals and, if the first and secondtest scores contain the discrepancy based on the predefined criteria; b)signal the first and second test resolvers to interactively determineand enter a joint score using the electronic communication link; and c)record the joint score.
 26. The system of claim 25 wherein thecomparison subsystem is further configured to track average scores ofthe first and second test resolvers and to prevent the first and secondtest resolvers from scoring a same test item if the first and secondtest resolvers each have average scores within predefined scoringcriteria.
 27. The system of claim 25 wherein the comparison subsystem isfurther configured to prevent the first and second test resolverterminals from receiving another answer to be scored until the first andsecond test resolvers have entered the joint score.
 28. The system ofclaim 25 wherein the comparison subsystem is further configured totransmit the test item to a third test resolver terminal to be scored bya third test resolver, if the first and second test resolvers do notenter the joint score within a predetermined time period.
 29. The systemof claim 23 wherein the comparison subsystem is further configured to:a) transmit the answer to a third test resolver terminal to be scored bya third test resolver, if the first and second test scores contain thediscrepancy based on the predefined criteria; and b) record a third testscore, as determined by the third test resolver, received from the thirdtest resolver terminal.
 30. The system of claim 23 wherein thetransmitting the answer to a third test resolver terminal by thecomparison subsystem further comprises the providing the third testresolver terminal with the first and second test scores.
 31. The systemof claim 23 wherein the transmitting the answer to a third test resolverterminal by the comparison subsystem further comprises the preventingthe third test resolver from viewing the first and second test scores onthe third test resolver terminal before the third test resolver recordsthe third test score.
 32. The system of claim 23, wherein the comparisonsubsystem is further configured to electronically link the first andsecond test resolver terminals when the first and second test resolversare located in different geographical locations.
 33. The system of claim23 wherein the comparison subsystem is further configured to selectivelyrecord, as the joint score, a greater of the first and second testscores, a lower of the first and second test scores, and an average ofthe first and second test scores.
 34. A computer-readable programstorage medium tangibly embodying a data package and associatedverification instructions executable by a computing system for verifyingtest answers between two or more test resolvers, the method comprisingthe steps of: a) electronically receiving an answer to a test question;b) electronically transmitting the answer to first and second testresolver terminals to be scored by first and second test resolvers; c)electronically receiving first and second test scores for the answer, asdetermined by the first and second test resolvers, from the first andsecond test resolver terminals; and d) electronically comparing thefirst and second test scores and for selectively recording a valuerelated to the first and second test scores if the first and second testscores did not contain a discrepancy based on predefined criteria. 35.The computer-readable program storage medium of claim 34 furthercomprising the step of electronically displaying the test question tothe first and second test resolvers at the first and second testresolver terminals.
 36. The computer-readable program storage medium ofclaim 34 wherein the step of electronically comparing the first andsecond test scores further comprises the steps of: a) providing anelectronic communication link between the first and second test resolverterminals and, if the first and second test scores contain thediscrepancy based on the predefined criteria, signaling the first andsecond test resolvers to interactively determine and enter a joint scoreusing the electronic communication link; and b) recording the jointscore.
 37. The computer-readable program storage medium of claim 36further comprising the step of tracking average scores of the first andsecond test resolvers and preventing the first and second test resolversfrom scoring a same test item if the first and second test resolverseach have average scores within predefined scoring criteria.
 38. Thecomputer-readable program storage medium of claim 36 wherein the step ofelectronically comparing the first and second test scores furthercomprises the step of preventing the first and second test resolverterminals from receiving another answer to be scored until the first andsecond test resolvers have entered the joint score.
 39. Thecomputer-readable program storage medium of claim 36 wherein the step ofelectronically comparing the first and second test scores furthercomprises the step of transmitting the test item to a third testresolver terminal to be scored by a third test resolver, if the firstand second test resolvers do not enter the joint score within apredetermined time period.
 40. The computer-readable program storagemedium of claim 34 wherein the step of electronically comparing thefirst and second test scores further comprises the steps of: a)transmitting the answer to a third test resolver terminal to be scoredby a third test resolver, if the first and second test scores containthe discrepancy based on the predefined criteria; and b) recording athird test score, as determined by the third test resolver, receivedfrom the third test resolver terminal.
 41. The computer-readable programstorage medium of claim 34 wherein the step of transmitting the answerto a third test resolver terminal further comprises the step ofproviding the third test resolver terminal with the first and secondtest scores.
 42. The computer-readable program storage medium of claim34 wherein the step of transmitting the answer to a third test resolverterminal further comprises the step of preventing the third testresolver from viewing the first and second test scores on the third testresolver terminal before the third test resolver records the third testscore.
 43. The computer-readable program storage medium of claim 34,wherein the step of electronically comparing the first and second testscores farther comprises the step of electronically linking the firstand second test resolver terminals when the first and second testresolvers are located in different geographical locations.
 44. Thecomputer-readable program storage medium of claim 34 wherein the step ofelectronically comparing the first and second test scores comprises thestep of selectively recording, as the value, a greater of the first andsecond test scores, a lower of the first and second test scores, and anaverage of the first and second test scores.